Fix source-location handling of user comments to be language independent #309

achidlow · 2024-09-10T06:01:09Z

SourceLocation now has a comment_lines field to indicate how many lines to fetch as user comments when annotating output source files.

This provides a language independent way of handling source locations that can optionally include comments, whereas the current approach was to strip them out when necessary, assuming # to be the comment delimiter.

Also line numbers for code that has no source data are no longer output - this mostly means the embedded lib functions, for which the line numbers were not particularly useful to begin with.

Merging of source locations when operations are combined in optimisation has also been improved, to only merge locations when they are overlapping or adjacent, thus avoiding including unrelated code in annotations.

Specifically source locations with self. members has also been improved to use the location of the usage, rather than declaration, this previously resulted in extremely large and unrelated annotations in certain situations, see for examples the changes in this PR to VotingRoundApp.approval.teal.

Non-user facing changes:

Improved SourceLocation validation, including making end_line non-optional (the existing behaviour of defaulting to line is retained, but is made explicit on the object).
Fixed a bug with MIR output alignment caused by virtual ops, which made the column heading misleading.
Fixed a bug with column locations when merging source locations.
Fixed a bug where column locations were retained when including source comments.
Fixed multiple places using user-comment locations when they should have been pointing at code only (e.g. IR labels, log output, ...)
Validate rather than resolve SourceLocation paths, to prevent inconsistencies on Windows CI/CD and also to guard against accidental relative path inclusion especially when deserializing AWST

github-actions · 2024-09-10T06:09:46Z

Coverage Report

File	Stmts	Miss	Cover	Missing
src/puya_lib
arc4.py	126	126	0%	1–332
bytes.py	9	9	0%	1–15
util.py	16	16	0%	1–33
src/puya
__main__.py	30	30	0%	1–42
arc32.py	68	2	97%	77, 103
artifact_sorter.py	53	1	98%	84
compile.py	147	8	95%	89–90, 132–135, 155–158, 288, 309
context.py	31	1	97%	35
errors.py	35	8	77%	40–42, 44–46, 50–51
log.py	199	41	79%	32–35, 69, 89, 108, 123–124, 166–168, 171–173, 175, 188–197, 219, 288–289, 298, 320–322, 335–349
main.py	35	35	0%	1–47
parse.py	87	5	94%	31, 40, 50, 56, 87
template.py	44	6	86%	18–19, 41–42, 58, 60
utils.py	199	18	91%	50, 67–68, 77–78, 86–87, 171, 177, 195–197, 204, 226, 249, 251, 274, 309
src/puya/awst
function_traverser.py	285	3	99%	76, 367, 373
nodes.py	992	44	96%	99, 103–106, 146, 150–153, 351, 537, 553, 587, 628, 657–658, 708, 736–737, 886, 908, 940, 968, 976, 981, 1136, 1221, 1237, 1286, 1359, 1430, 1484, 1488, 1734, 1745, 1747, 1752, 1760, 1765, 1770, 1779, 1784, 1789
serialize.py	57	1	98%	20
to_code_visitor.py	439	7	98%	131, 315, 365, 621, 646, 674, 678
txn_fields.py	98	1	99%	48
wtypes.py	294	17	94%	174–178, 188–191, 208, 226, 250, 305, 319, 353, 424, 469
src/puya/awst/validation
base_invoker.py	47	4	91%	55, 62, 69–73
inner_transactions.py	181	1	99%	160
labels.py	30	8	73%	25–27, 32, 36–41
scratch_slots.py	39	4	90%	19, 33, 49, 51
src/puya/ir
arc4_router.py	312	20	94%	110, 173, 229, 307, 355, 424, 433, 491–492, 504, 509, 514, 519, 524, 529, 549–553, 737, 762, 799
avm_ops.py	315	1	99%	46
avm_ops_models.py	48	3	94%	21, 30, 38
context.py	107	8	93%	78, 85, 88, 90, 128–129, 141, 147
main.py	299	12	96%	74–80, 90–96, 122, 163, 221, 565
models.py	529	22	96%	74, 182, 189, 338, 408–409, 414, 420–424, 437, 481, 510, 566, 612, 692, 708, 749, 752, 759, 762, 851–852
ssa.py	130	3	98%	51–52, 150
to_text_visitor.py	152	8	95%	123, 128, 212, 219–224
types_.py	101	10	90%	50, 57, 91–95, 116, 152, 157–159
visitor.py	129	17	87%	145, 193, 205, 220, 223, 236, 239, 245, 248, 259, 262, 265, 268, 271, 274, 277, 280
visitor_mutator.py	105	2	98%	174–175
vla.py	72	1	99%	87
src/puya/ir/builder
_utils.py	58	3	95%	175–177
arc4.py	504	25	95%	93, 129–134, 375, 400, 403, 461–464, 719–720, 966, 990, 1071, 1078, 1114, 1162, 1172, 1227, 1272, 1291, 1311, 1372–1379
assignment.py	83	6	93%	51, 98, 112, 188, 206, 222
blocks.py	140	7	95%	55, 92–96, 158, 166, 231
bytes.py	64	14	78%	13–45, 129
callsub.py	76	6	92%	30–31, 51, 101–102, 123
flow_control.py	95	1	99%	56
iteration.py	198	5	97%	91–92, 107, 144, 206
itxn.py	276	19	93%	124–125, 127, 141, 189, 210, 235–236, 565, 589–590, 603–612
main.py	545	65	88%	106, 247, 251, 256–274, 279–297, 352, 376, 400–401, 439, 627, 649, 662–663, 719, 744, 781–783, 794, 859, 937, 950, 983, 1038, 1041, 1049, 1052, 1060, 1089, 1100, 1109, 1188, 1205, 1248–1258
storage.py	83	2	98%	101, 154
src/puya/ir/destructure
coalesce_locals.py	101	19	81%	119, 128–129, 132–135, 138–147, 163–166
parcopy.py	84	2	98%	47, 83
src/puya/ir/optimize
_utils.py	14	2	86%	17–19
collapse_blocks.py	92	5	95%	65–69
compiled_reference.py	79	5	94%	53, 87, 158–163
control_op_simplification.py	104	15	86%	47–48, 92–121, 171, 245–252
inner_txn.py	36	1	97%	38
intrinsic_simplification.py	483	33	93%	76, 166–168, 173, 255, 267, 302, 313–314, 338–339, 363, 430, 568, 596, 617, 619, 634, 674, 697, 727, 733, 735, 737, 742, 744, 746, 748, 750, 802–803, 810
main.py	87	2	98%	116–117
src/puya/ir/validation
_base.py	29	1	97%	24
compile_reference_validator.py	20	3	85%	24, 30, 37
min_avm_version_validator.py	11	1	91%	15
op_run_mode_validator.py	19	5	74%	19–29
src/puya/mir
annotaters.py	159	2	99%	62, 212
builder.py	140	10	93%	139, 248–249, 302, 305, 308, 311, 314, 317, 320
models.py	328	16	95%	40, 52, 68, 77, 88, 99, 110, 150, 260, 291, 328, 340, 365–371
output.py	63	2	97%	29, 33
stack.py	250	19	92%	72, 119, 135, 146, 157, 167, 184, 201, 211, 230, 250, 256, 258, 297, 310, 334, 366, 371, 409
src/puya/mir/stack_allocation
baileys.py	205	3	99%	31, 330–334
frame_allocation.py	80	2	98%	21, 68
koopmans.py	61	1	98%	55
peephole.py	100	6	94%	75, 89, 99, 101, 103, 145
src/puya/teal/optimize
peephole.py	112	2	98%	139, 145
repeated_rotations.py	42	1	98%	13
repeated_rotations_search.py	88	6	93%	34, 40–41, 57, 67–68
src/puya/ussemble
build.py	73	10	86%	37, 52–53, 58–61, 65–68, 85–86, 115
op_spec_models.py	22	1	95%	20
optimize.py	76	1	99%	77
output.py	115	1	99%	71
validate.py	17	1	94%	19
src/puyapy
__main__.py	37	1	97%	153
client_gen.py	115	15	87%	55–56, 65–69, 73, 165–166, 174–175, 190, 194–195, 204
compile.py	151	22	85%	59–68, 81, 162–163, 174, 181–182, 199–209, 219–221, 226, 242
parse.py	179	13	93%	49–50, 86, 154–159, 163, 261, 265–266, 313, 325, 350
utils.py	21	5	76%	16–17, 25–28
src/puyapy/awst_build
arc4_client.py	70	19	73%	45, 54, 60, 63, 66, 72, 75, 78–83, 86, 89, 92, 95, 98, 101, 104, 107, 110
arc4_utils.py	324	48	85%	45, 48–49, 51, 92, 104–106, 133, 144–148, 164, 166, 169–170, 193, 234–236, 245, 250, 255–256, 280, 284, 296, 303, 305, 319–320, 323–327, 332–333, 339–342, 350, 368, 380, 388–389, 401, 409, 510, 534
arc32_client_gen.py	68	2	97%	34, 130
base_mypy_visitor.py	127	39	69%	70–76, 94, 102–115, 129, 131, 133, 145, 150, 154, 157, 160, 166, 188, 191, 194, 201, 205, 208, 211, 215, 233, 237, 241, 245, 249, 253, 257, 261, 265, 269, 273, 277, 281
context.py	236	47	80%	66, 96, 99, 109–110, 128–129, 169, 229, 234, 240–244, 249, 258, 260, 263–265, 267, 274, 276, 287–288, 293–295, 298, 318, 342–343, 355, 369, 372–384
contract.py	238	27	89%	182, 224–225, 227–230, 242, 254, 345, 349, 371, 374, 386, 394, 397, 400, 403, 406, 409, 412, 415, 418, 421, 435, 458–464
contract_data.py	58	1	98%	37
intrinsic_factory.py	32	3	91%	47, 56, 65
intrinsic_models.py	40	1	98%	49
main.py	52	1	98%	34
module.py	402	57	86%	115, 129–131, 146–147, 154, 163–164, 172–176, 194–198, 207, 245–246, 257, 279–282, 292–294, 300, 317–320, 333, 367, 374, 396–397, 420–425, 479–480, 508, 519, 522, 528, 534, 544, 550, 553, 565, 568, 590, 610, 615, 619, 623–626, 651, 694, 702, 704
pytypes.py	488	52	89%	85–87, 101–102, 142, 158–164, 187, 207, 282, 291, 310, 329, 333, 349–350, 382, 475–477, 491–492, 561–562, 692, 703–704, 749–750, 755, 804–805, 826–827, 956–957, 981, 1009, 1043–1045, 1071, 1098, 1108–1109, 1145–1147
subroutine.py	650	55	92%	139, 266, 272, 330–333, 339, 391, 398, 401–407, 472, 633, 635–636, 654, 656, 666–667, 676–677, 681, 702, 775, 782, 802–803, 885, 916–917, 934, 956, 962, 1050, 1058, 1069, 1094, 1199–1200, 1221, 1231, 1238, 1247, 1263, 1273–1274, 1282, 1301, 1304, 1307, 1310, 1313, 1333–1335
utils.py	185	24	87%	35, 52–56, 74, 109–110, 112, 156–157, 208, 216, 221, 234–238, 243–246, 288, 296, 339
src/puyapy/awst_build/eb
_base.py	128	19	85%	52, 57–59, 64, 71, 76, 81–83, 142, 153, 177, 182, 187, 192, 203, 218, 223–225
_bytes_backed.py	48	2	96%	30–31
_expect.py	125	18	86%	26, 37, 78–81, 93–96, 99, 158–159, 220, 233–236
_literals.py	138	29	79%	43, 72, 91, 120, 128, 142, 146, 150–156, 166–180, 185
utils.py	44	1	98%	96
array.py	26	9	65%	23, 28–33, 43, 49
biguint.py	96	6	94%	56, 98, 135, 150–151, 153
bool.py	50	8	84%	37–41, 57, 68, 83
bytes.py	169	18	89%	102–103, 130–131, 136–137, 143–144, 147, 155, 198, 233, 265, 269, 286–287, 302–303
compiled.py	112	13	88%	56, 61, 67–68, 165–171, 206–210, 235
conditional_literal.py	126	35	72%	94, 98, 152, 156–159, 168–170, 179–182, 193–196, 205, 209, 213–216, 231–243, 252–253
contracts.py	83	6	93%	58, 64, 76, 101, 122, 126
dict.py	27	5	81%	25, 33–35, 39
ensure_budget.py	31	1	97%	47
interface.py	84	3	96%	302–304, 308
intrinsics.py	97	6	94%	43, 62, 69, 82, 89, 160
log.py	42	4	90%	45–46, 51, 60
logicsig.py	15	1	93%	26
none.py	27	1	96%	38
string.py	145	13	91%	72, 115–116, 135, 139, 182, 189, 193, 205, 279–281, 301
struct.py	16	5	69%	14–16, 25, 31
subroutine.py	81	17	79%	46, 50–53, 68, 71–78, 93, 101–105, 107–110, 115
template_variables.py	37	2	95%	30, 58
tuple.py	285	13	95%	76, 148, 154, 189–190, 196, 273–274, 393, 421, 432–433, 496
uint64.py	110	5	95%	57, 118–119, 167–168
uint64_enums.py	40	2	95%	41, 46
unsigned_builtins.py	152	21	86%	73, 80, 104, 128, 132, 136, 140, 148, 152, 156, 160, 164, 174, 178, 184, 195, 201, 240, 272, 284, 296
src/puyapy/awst_build/eb/arc4
_base.py	91	3	97%	186–189, 200
_utils.py	131	10	92%	78–79, 106, 111, 124, 164–167, 207, 211, 225
abi_call.py	321	20	94%	122, 128, 132, 156, 226, 243–244, 325, 342, 400, 428, 458, 481–482, 540, 577, 649, 734–735, 752
address.py	77	3	96%	57, 117–118
bool.py	57	3	95%	44, 86–87
dynamic_array.py	127	10	92%	57, 126–127, 147, 149, 154, 228, 249, 255–258
dynamic_bytes.py	68	3	96%	97–99
emit.py	36	1	97%	38
static_array.py	66	1	98%	45
string.py	100	7	93%	54–55, 103, 126, 131–134
struct.py	49	1	98%	49
tuple.py	94	15	84%	50–52, 92–95, 98–99, 135–138, 143, 147–148, 158, 168
ufixed.py	70	2	97%	43, 102
src/puyapy/awst_build/eb/reference_types
account.py	81	2	98%	65, 176
application.py	45	1	98%	40
asset.py	65	1	98%	48
src/puyapy/awst_build/eb/storage
_common.py	69	3	96%	107, 122–123
_storage.py	100	17	83%	76, 84, 88, 92, 96, 100, 104, 108, 112, 122, 126, 130, 134, 140, 151, 157, 169
_value_proxy.py	55	7	87%	38, 42, 50, 54, 91, 99, 103
box_map.py	143	1	99%	186
global_state.py	138	5	96%	108–109, 115, 171–172
local_state.py	148	11	93%	103–104, 108, 157, 161, 165, 175, 179, 203, 290, 314
src/puyapy/awst_build/eb/transaction
base.py	39	2	95%	23, 43
group.py	58	1	98%	51
inner.py	48	2	96%	88–89
inner_params.py	81	5	94%	69, 79, 83, 143, 145
itxn_args.py	60	1	98%	72
TOTAL	21464	1667	92%

Tests	Skipped	Failures	Errors	Time
1263	3 💤	0 ❌	0 🔥	6m 54s ⏱️

src/puyapy/awst_build/eb/contracts.py

…access

- output all source locations in MIR, even if they're synthetic

… lines

…-intervals are merged

…stencies on Windows CI/CD and also to guard against accidental relative path inclusion especially when deserializing AWST

achidlow added 2 commits September 10, 2024 11:37

remove unused functions

cbedf09

fix MIR writer alignment bug

eb17c59

achidlow force-pushed the comment-locations branch from 1455467 to 5cb4304 Compare September 10, 2024 06:40

achidlow enabled auto-merge (rebase) September 10, 2024 06:51

daniel-makerx reviewed Sep 10, 2024

View reviewed changes

src/puyapy/awst_build/eb/contracts.py Outdated Show resolved Hide resolved

achidlow added 17 commits September 10, 2024 18:20

fix: when accessing a member of self, use the source location of the …

2c67c4a

…access

- don't output line/file numbers for embedded lib functions in TEAL

5baf452

- output all source locations in MIR, even if they're synthetic

refactoring / error message casing

a916ff5

remove unnecessary SourceMeta

21c452f

minor refactor/simplify

231c3a7

fix column locations when expanding source to include/exclude comment…

9b2c112

… lines

make end_line non-optional

c2daf02

fix bug in SourceLocation "addition"

4ab8a8b

remove unused optional argument

74d2f84

improved SourceLocation validation

8cfcb38

remove comment Python coupling

dcffc85

fix user-comment locations being included in log output locations

18b83a9

fix block line locations in IR labels

87a3098

removing user-comment lines from MIR output

ba4cf15

improve SourceLocation merging so that only adjacent/overlapping line…

dfebc41

…-intervals are merged

simplify traceback logging

2f5e3d2

validate rather than resolve SourceLocation paths, to prevent inconsi…

6f29ea3

…stencies on Windows CI/CD and also to guard against accidental relative path inclusion especially when deserializing AWST

achidlow force-pushed the comment-locations branch from 5cb4304 to 6f29ea3 Compare September 10, 2024 10:44

daniel-makerx approved these changes Sep 10, 2024

View reviewed changes

achidlow merged commit 1d7b06f into main Sep 10, 2024
4 checks passed

achidlow deleted the comment-locations branch September 10, 2024 11:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix source-location handling of user comments to be language independent #309

Fix source-location handling of user comments to be language independent #309

achidlow commented Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024 •

edited

Loading

Fix source-location handling of user comments to be language independent #309

Fix source-location handling of user comments to be language independent #309

Conversation

achidlow commented Sep 10, 2024 • edited Loading

github-actions bot commented Sep 10, 2024 • edited Loading

achidlow commented Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024 •

edited

Loading